Multi-Level Compositional Reasoning for Interactive Instruction Following

Robotic agents performing domestic chores by natural language directives are required to master the complex job of navigating environment and interacting with objects in environments. The tasks given often composite thus challenging as completing them require reason about multiple subtasks, e.g., bring a cup coffee. To address challenge, we propose divide conquer it breaking task into subgoals attend individually for better navigation interaction. We call Multi-level Compositional Reasoning Agent (MCR-Agent). Specifically, learn three-level action policy. At highest level, infer sequence human-interpretable be executed based on instructions high-level policy composition controller. middle discriminatively control agent’s alternating between various independent interaction policies. Finally, at lowest manipulation actions corresponding object masks using appropriate Our approach not only generates human interpretable but also achieves 2.03% absolute gain comparable state arts efficiency metric (PLWSR unseen set) without rule-based planning or semantic spatial memory. code is available https://github.com/yonseivnl/mcr-agent.